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Abstract 

As more and more activities are performed in space, there will be a greater demand placed on 
the information handling capacity of people who are to direct and accomplish these tasks. A 
promising alternative to full-time human involvement is the use of semi-autonomous, intelligent 
robot systems. To automate tasks such as assembly, disassembly, repair and maintenance, the 
issues presented by environmental uncertainties need to be addressed. These uncertainties are 
introduced by variations in the computed position of the robot at different locations in its work 
envelope, variations in part positioning, and tolerances of part dimensions. As a result, the robot 
system may not be able accomplish the desired task without the help of sensor feedback. 
Measurements on the environment allow real time corrections to be made to the process. This 
paper presents a design and implementation of an intelligent robot system which inserts printed 
circuit boards into a card cage. Intelligent behavior is accomplished by coupling the task 
execution sequence with information derived from three different sensors: an overhead three- 
dimensional vision system, a fingertip infrared sensor, and a six degree-of-ffeedom wnst-mounted 
force/torque sensor. 


1 . Introduction 

Robots are still far from the flexible automation tool they were envisioned to be. Robots in the 
present day generally operate with minimal sensing and use a control strategy based on open loop 
positioning. Thus the reproducibility of the task depends upon the repeatability of the robot motion 
and the experimental setup used. A reliable robotic system has to be able to accommodate 
uncertainties in the environment due to poor repeatability of the robot, changes in the workspace, 
and variations in the position, orientation, and dimensions of the workpieces. For this reason 
feedback from the workspace is required and therefore sensors have to be used. The sensory 
information is used to aid the robotic system in accomplishing the desired task. Vision, tactile, 
force/torque, proximity, and crossfire sensors can be used. 

In order for robots to perform tasks like assembly, repair, or maintenance, the robot has to be 
programmed using motion control primitives. Robotic assembly tasks can be programmed in 
various languages, some of them incorporating both the robot motion primitives and the sensory 
interactions. These languages include AL by Mujtaba and Goldman [1], LM by Latombe and 
Mazer [2], and AML by Taylor, Summers and Meyer [3]. Work on fine motion planning involving 
sensor-guided motions to achieve part mating has been presented by Brooks [4j. A solution to the 
problem of mating two parts requiring sensor based strategies to deal with geometric uncertainties 
has been presented by Dufay and Latombe [5]. Their strategy to handle assembly forces is based 
on threshold monitoring, wherein the robot motion is earned out until a sensory-based condition is 
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met. Once the condition has been met, the motion is stopped, even if the desired goal has not been 
achieved. 

In the literature, several approaches for controlling the robot based on the forces measured 
from the sensors present on the manipulator have been suggested. A survey of these strategies is 
given in Cutkosky and Wright [6]. Two of these approaches are based on compliance, one passive 
and the other active. Passive compliance uses compliant tools and is described in Nevins et al. [7]. 
Active compliance is achieved by using servo loop compensators and is described in Nevins et al 
[7] and Nevins and Whitney [8]. 

In this paper, a strategy based on fuzzy set theory is used to interpret the forces and torques 
generated during the printed circuit board insertion process. It is quite cumbersome to derive a 
mathematical model to describe an assembly task, such as the one described here. If such a model 
were to be developed, it would be specific to the details of the particular task. Goldenberg and 
Bazerghi [9] present an example of a method using a mathematical model for a peg-in-hole 
problem. The fuzzy approach uses approximate relationships instead of a mathematical model 
which uses absolute, numerical quantities. In the real world of robotic assembly, the goals, 
constraints and consequences of robot actions are not precisely known and, therefore, cannot be 
modelled exactly. Thus, the decisions have to be made by means of an inference mechanism that 
can handle uncertain and imprecise knowledge. According to Zadeh [Yager et al., 10], if the gap 
between human intelligence and machine intelligence is to be narrowed, machines should acquire 
the ability to manipulate fuzzy concepts and to respond to fuzzy instructions. Fuzzy logic, based 
on fuzzy set theory, is used for approximate reasoning about the insertion task. 

The target task presented in this paper is part of a hierarchical planning and execution system. 
This system maps user-specified three-dimensional part assembly tasks into various target robotic 
workcells, and executes these tasks efficiently using manipulators and sensors available in the 
workcell. This system was researched and developed in the Robotics and Automation Laboratories 
at Rensselaer Polytechnic Institute. Details of this system are presented in Kelley and Moed [11]. 
As part of the hierarchy, the vision controller and the fuzzy insertion controller are each on-line, 
independent processes that are experts in accomplishing specific tasks. For this reason, these two 
controllers are called specialists. The specialists developed for this work are "plugable" modules 
which can be executed together with other existing specialists to perform a multitude of assembly 
tasks within the hierarchical structure. 


2 . Experimental Setup 

The experimental set up for the printed circuit board insertion task consists of a robot, a host 
computer, three basic sensors, and assembly fixtures. The first sensor is a 3D vision system 
comprised of two CCD cameras and four lights suspended above the robot workcell. The cameras 
are calibrated with respect to the robot’s coordinate system using a method developed by 
Yakimovsky and Cunningham [12, Kwak 13]. The 3D vision system is used to find the gross 
location of objects in the workcell. The second sensor, a finger tip mounted crossfire sensor, 
complements the vision system by providing more accurate position information. The third sensor, 
a wrist-mounted force/torque sensor, monitors the insertion process. 

The printed circuit boards are placed in the robot workcell in a fixture called the pick-up rack. 
The boards are to be inserted in the card cage into slots called the insertion slots. The task starts by 
taking a picture of the workcell with each camera. In the workcell, the general location of the 
printed circuit board pick-up rack and the insertion slots is known a priori. Thus, in each image, 
separate windows can be defined around the nominal locations of the pick-up rack and the 
insertion slots. To save time, all image processing is confined to these windows. The windows are 
large enough so that small variations in the position and orientation of the pick-up rack or insertion 
slots are accommodated. A binary image of each window is created by thresholding the image. 
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Blob labelling and moment generating techniques are used to find a good grasping location on the 
printed circuit boards. Since the printed circuit boards are standing vertically, it is desirable to 
grasp them at the midpoint of the top edge. Because the cameras are not looking directly at the 
edges of the boards, the resulting image of a board is not a line (which might correspond to the top 
edge of a board). Instead, the board image is a parallelogram corresponding to a two dimensional 
projection of the board on the camera image plane. The long edges of the parallelogram correspond 
to the top and bottom edges of the printed circuit board. The first order moments are used to find 
the centroid of this parallelogram which represents a point in the center of the board. As previously 
mentioned, the center point of the top edge is a desirable location for grasping the printed circuit 
boards. Since the cameras can be in any position, the top edge of the parallelogram in the image 
does not necessarily correspond to the top edge of the printed circuit board. Thus, ray casting is 
used to determine which edge of the parallelogram represents the top edge of the board. A ray is 
generated from the lens of the camera through the centroid and is projected onto the x-y plane of 
the work surface. If the y-component of the projection is negative, then the camera is located above 
and to the left of the printed circuit board. In this case it can be inferred that the top edge of the 
parallelogram corresponds to the top edge of the board. If the y-component is positive, then the 
camera is above and to the right of the printed circuit board. In this case it can be inferred that the 
bottom edge of the parallelogram corresponds to the top edge of the printed circuit board. The 
binary image is scanned from the centroid towards the top edge of the board to determine the image 
coordinates of the grasping location on the board. Using the parameters from the camera 
calibration model, the image coordinates of the grasping location are transformed to robot 
coordinates. A similar approach is used to find the robot coordinates of the insertion slots (the only 
difference is that the slot height is known a priori). The angular rotation of the boards and the 
insertion slots in the image is determined using the information available from the second order 
moments. Again the calibration model is used to transform the orientation to the robot coordinates. 
Due to limitations of the camera calibration routines and inaccuracies introduced by thresholding 
the image, these robot coordinates are treated as only a first estimate. 

The crossfire sensor is simply constructed from an infra-red emitting diode and a photo- 
detector. Initially, the robot is directed by the supervisor to move the gripper above the grasping 
location calculated by the vision system. Then the gripper is moved straight down in small 
incremental steps. At each step, the finger-tip crossfire sensor is polled. When die infrared beam is 
broken by the top edge of the board, the photo-detector output changes drastically. Thus a 
reasonably accurate value for the z-coordinate of the top edge of the board is determined. Next the 
crossfire sensor is used to find the exact center of this edge. The gripper is moved along the length 
of the board until it finds one side. The process is repeated in the opposite direction to find the 
other side of the board. From this, the x-, y,- and z-coordinates of the grasping location are 
calculated and the gripper is moved to that position and the fingers are closed. 

The printed circuit board is picked up by the robot and moved to a position above the insertion 
slot. The printed circuit board is lowered to within a few centimeters above the height of the slot 
guides. At this point the force/torque sensor is zeroed and the fuzzy controller is used to place the 
board into the slot guides and then insert it in the slot. 

The fuzzy controller monitors a six-component force/torque vector which relate to changes to 
be made in the position and orientation of the robot. If all the components of this vector are zero, 
the gripper is moved in the negative z-direction in small steps which lowers the board into the 
guides. Otherwise, the robot is moved to the new position and orientation which is obtained by 
adding a correction obtained from the fuzzy controller to its current values. This is repeated until 
the board is well into the guides. Then the board is moved in the negative z-direction in larger 
steps. In this phase the weighting of the vector returned from the fuzzy controller is reduced since 
the board is already in the guides. The board is inserted until it is close to the top of the slot. At this 
point the z-force is monitored in order to seat the board into the slot without damaging it. Once the 
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board has been inserted the gripper is opened and moved up vertically to clear the height of the 
guides. The robot is then directed to pick up another board and repeat the entire process. 

3. Use of Fuzzy Logic 

The design and synthesis of conventional controllers is based on the mathematical model of 
the plant and involves quantitative and numeric calculations. With the advances in the area of fuzzy 
logic and linguistic reasoning, fuzzy controllers are being used to control systems and replace the 
human operator an integral part of the control process. These controllers use strategies expressed 
as linguistic statements, that resemble human decision making. Holmblad and Ostergaard [14] 
describe the application of fuzzy logic to the computer control of a rotary cement kiln. They 
conclude that fuzzy logic is a practical and realistic alternative to traditional means of implementing 
control strategies based on mathematical models. Fuzzy logic makes reasoning in the real world 
possible by providing the ability to deal with a continuous range of values rather than just true or 
false. Elements in fuzzy sets may belong only partially to a set, in contrast to traditional set theory 
where elements either belong to a set or not 

When an assembly task is performed by a human, the reasoning that comes into play is often 
of the form: 


IF <sensed condition> THEN ccontrol action> . 

Depending upon the parts being assembled, the sensed conditions could be different. In the 
case of inserting a card into a card cage, the sensed conditions could be of the form: 

"The card is not completely aligned with both the guides" 


or 


"The card is not being pushed in vertically." 

The information about the alignment of the card can be obtained visually. The information about 
the direction of application of the pushing force can be obtained either visually or through contact 
sensing. The control actions, likewise, could be of the form : 

"Reorient the card slightly to bring it in line with the guides" 


or 


"Change the direction of application of the force a little to eliminate the jamming." 

When a robot performs the task of a printed circuit board insertion, sensors are needed to 
obtain the information regarding the process. In this experiment, force/torque information is used 
to help the robot insert a printed circuit board into a slot whose location is determined using the 
overhead camera system. The reasoning involved in the robotic insertion task is of the form of the 
IF-THEN expression as given above. The sensed conditions involve terms related to the changes 
in the forces and torques observed during the process. The sensed conditions are of the form: 

"The x-torque is positive big" 


or 


"The y-torque is negative small," 
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and so forth. The logic value of a condition in ordinary Boolean logic is restricted to true or false 
(0 or 1). In fuzzy logic, the logic value is a measure of the fulfillment of the condition and can take 
any value in the interval [0, 1]. Fuzzy logic is used to express each of the terms by a unique fuzzy 
membership function and thus establish a value in the interval [0, 1] for a given condition. The 
forces and torques read from the sensor are scaled in such a fashion such that the maximum and 
minimum values are -100 and 100 units of force (uf = 0.2 oz) respectively. The scaling factors are 
formulated from the signatures obtained during the trial insertion processes. Thus each sensor 
reading is mapped onto a universe of discourse of -100 to 100 over which the relevant fuzzy sets 
are defined. A parametric representation is used to represent the fuzzy sets. For example, if a fuzzy 
set is defined as (a, b, c, d), this means that the membership value is 1 from a to b and goes from 1 
to 0 along a straight line on either side from a to a - c and from b to b + d (c and d being positive). 
The representation for "negative big" is (-100, -70, 0, 20). Figure 1 shows the fuzzy sets used in 
the course of this experiment. 



nb - negative big 
nm - negative medium 
ns - negative small 
ps - positive small 
pm - positive medium 
pb - positive big 


Figure 1 . Fuzzy sets used in the card insertion experiment. 


The forces and torques generated during a robotic assembly task reflect the status of the 
process: jamming, misalignment, seating, for example. These forces and torques serve as 
characterizing signatures of the process. Using statistical classification techniques, Fullmer [15] 
uses these signatures to classify the assembly process as either acceptable or unacceptable. In this 
paper, the force and torque signatures form a basis for the development of the fuzzy rules that are 
used to provide real time corrective measures to accomplish the insertion task. The 6-D 
force/torque signatures of an assembly process can be obtained by recording the forces and torques 
as the assembly proceeds. For a typical "card-into-a-slot" insertion, the signatures are obtained 
with respect to the position along the insertion path. In this case, the z-position of the card in the 
guides is used. These signatures are affected by the vibrations of the robot, and have a component 
that is related to the motion of the robot. Figure 2 shows these signatures for the robot going 
through the insertion motion without a card in the gripper. Thus, the exact values of the forces and 
torques generated might not be the same for two successive insertions of the same card into the 
same slot. It can be assumed, however, that there will be basic features and trends of the 
force/torque pattern which are common to successful insertions. The basic features may be 
parameters associated with the peaks and valleys in the signatures. In case of some uncertainty in 
the environment, it is difficult to classify these signals exactly and thus recognize the situation at 
hand. It is almost impossible to take into account all the possibilities that can arise during an 
assembly task and store them. The fuzzy controller implementation is able to cope with the 
uncertainties and to successfully interpret the signatures. 

After looking at the geometry of the "card-in-a-slot" problem, a number of heuristics capturing 
the fuzzy reasoning process were generated. Every heuristic is represented as a condition/action 
fuzzy rule. The condition, or the left hand side of a rule, tests to see whether the rule is applicable 
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to the situation at hand. The action, or the right hand side of the rule, consists of a list of actions to 
be performed if the rule is applicable. 



Figure 2. Typical force/torque signatures obtained by performing the insertion motion without a 
card in the gripper. 


4 . Experimental Results 

In this section, the test results of the "insert_board" specialist are presented. First, the PC 
board is located using the "3D_vision" specialist and the position and orientation of the board is 
estimated. This information is provided to the robot controller and the gripper is moved to a 
position above the desired board. Because of the limited resolution of the overhead vision system, 
the crossfire sensor in the finger tips is used to determine the precise edge locations of the board. 
The position and orientation of the insertion slot are also determined using the vision system. 
Next, the board is picked up at the center of its top edge and the arm positioned above the slot 
where the board is to be inserted. Finally the board is inserted into the guides using the fuzzy 
controller with feedback from the wrist force/torque sensor. 

The fuzzy rules are written based on the force/torque signatures obtained from repeated trials. 
Figure 3 shows two of the signatures for a perfect insertion. The increase in the z-force indicates 
the seating of the board into the slot. Also, the torque shows an increase in magnitude as the card 
is lowered into the guides because of friction and deviations in the robot motion relative to a 
straight line when moved in cartesian coordinates. 

Consider a simple problem that might be faced by the insert_board specialist as shown in 
Figure 4. In this case the knowledge that is needed to insert the board into the guides is as follows: 

1) A medium change in the z-force and large changes in the x- and y-torques as the board is 
lowered indicate that one of the comers of the board is caught on one side of the guide. 

2) No appreciable changes in the forces and torques as the card is lowered into the guides 
indicate that the insertion is proceeding normally. 

3) A large change in the z-force after the gripper has moved approximately the height of the 
board indicates that the board is being seated into the slot. 
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Figure 3. Typical force/torque signatures obtained by performing a perfect insertion. 


Gripper 





Figure 4. Typical near-miss situation to be handled by the insert board specialist and orientation of 
the wrist force/torque sensor coordinate system. 


A typical rule written in C is shown below. The gist of the rule is that if the scaled value of the 
x-torque (mapxte) is negative big (nb) and the scaled value of the y- torque (mapyte) is positive big 
(pb) then the output, ( outset ) should be negative big (nb). 

Rule: Condition min = findmin( nb, mapxte, pb, mapyte) 

Action truncset (nb, min, tempset) 

maxfn (outset, tempset) . 

Findmin examines the fuzzy set nb at the point mapxte and the fuzzy set pb at mapyte , and returns 
the minimum of the two. Then the minimum value min is used to truncate the fuzzy set, nb, and 
create tempset, a temporary set. Maxfn is used to accumulate the effect of the rules that influence 
that particular output. It takes the maximum of the two sets outset and tempset and stores it in 
outset. In this case the output variable is a change in the x-position of the robot arm. This is 
represented by delta-x which is obtained by defuzzifying the resulting set outset. 


369 



Delta-0 (deg) Delta-Y (nun) 




Control Iteration 


Control Iteration 



Control Iteration 


Control Iteration 


0.6 
0.4 

0.2 
o.o 
- 0.2 
- 0.4 
- 0.6 

0 10 20 30 40 50 60 70 80 0 10 20 30 40 50 60 70 80 

Control Iteration Control Iteration 

Figure 5. Results of a successful insertion process. 



Figure 5 shows the results of a successful insertion process. The graphs show the 
modifications made to the x- and y-positions of the gripper and its rotation about the z-axis in the 
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robot coordinate system. Also depicted are the x- and y-torques and the z-force. The control 
iteration axis in the graphs refers to every point where a decision is made. The following strategy 
is employed: If the delta-x, delta-y, and the delta-o are each below a given threshold, the board is 
moved in a fixed increment towards the insertion slot. If any reading is above its threshold, 
corrective action is taken instead. As shown in the figure, between control iterations 15-25 the 
near-miss situation described in Figure 4 is encountered. The delta-x and delta-y graphs show the 
corrective measures taken. The change in the rotation shown by delta-o relieves the torques by 
aligning the board. During insertion, the card became skewed due to slippage in the gnpper. 1 he 
iterations 65-80 show the activity in delta-x and delta-y which corrects this skew. From control 
iterations 60 onwards the delta-o corrections are ignored by the controller to best avoid 
oscillations. This explains the increase in the x- and y-torques during this period. The increase in 
the z-force at iteration 83 is due to the final insertion of the board into the insertion slot. 


5. Conclusions 

A robotic assembly system has been presented which couples the task sequence of the robot 
with information from different sensors. This coupling enables the system to handle uncertainties 
encountered during task execution. In the example task only the nominal position of the printed 
circuit boards and the insertion slots is known. A 3-D vision system determines the approximate 
position and orientation of the boards and the insertion slots. It also determines a proper grasping 
location on the board which is refined with the help of a fingertip crossfire sensor. A six degree- 
of- freedom force/torque sensor is used to sense the forces and torques generated during the 
assembly process. This paper described the application of fuzzy logic techniques to characterize 
relationships between the assembly objects and the data from the force/torque sensor. The fuzzy 
approach uses experienced-based approximate relationships instead of using a precise mathematical 
model of the insertion process. Data collected from a typical execution was shown in order to 
describe the information received from the sensors. A typical fuzzy control rule for this task was 
also included and explained. By using a fuzzy controller, printed circuit boards were successfully 
inserted into target insertion slots. The modeling and controller assumptions made by fuzzy set 
theory were validated by the repeated success of this assembly task over many trials. 
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